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Motivated by the absence of cooperative Jahn- Teller effect and of magnetic ordering in LiNi02, a 
layered oxide with triangular planes, we study a general spin-orbital model on the triangular lattice. 
A mean- field approach reveals the presence of several singlet phases between the SU(4) symmetric 
point and a ferromagnetic phase, a conclusion supported by exact diagonalizations of finite clusters. 
We argue that one of the phases, characterized by a large number of low-lying singlets associated to 
dimer coverings of the triangular lattice, could explain the properties of LiNi02, while a ferro-orbital 
phase that lies nearby in parameter space leads to a new prediction for the magnetic properties of 
NaNi02. 



I. INTRODUCTION 

The Mott insulators LiNi02 and NaNi02 are isostruc- 
tural and isoelectronic, but they have completely differ- 
ent phase diagrams. The complicated nature of these sys- 
tems arises from an interplay of the dynamical frustration 
of spin-orbital models with the geometrical frustration 
of the triangular lattice which is the essential structural 
unit. We will show that by a modest change of parame- 
ters, a great variety of phases can be derived. 

The crystal structure can be envisaged as a sequence 
of slabs of edge sharing octahedra of oxygen ions. 
Metal ions sit at the centers of octahedra. There are two 
kinds of slabs: in A slabs, at every center of octahedra 
there is a Ni^"*", whereas in the B slabs, one finds cither 
Li+ or Na"*" ions. A and B slabs alternate (see Fig. 
The Ni ions form well-separated triangular planes. 

It is useful to start with the idealized geometry of a 
cubic system. Neglecting the inequivalence of Ni and Li 
sites, and assuming perfect oxygen octahedra, the octa- 
hedral centres would form a simple cubic lattice. The 
slabs of the original structure would be perpendicular to 
the 111 direction. Within a slab the Ni-O-Ni bond an- 
gles would be 90°, resulting in important consequences 
for the effective exchangai. 

There are two sources of deviation from cubic symme- 
try: a) Ni and Li/Na sites are inequivalent, which leaves 
us with one (instead of four) C3 axis. Even if the octahe- 
dra were undistorted, Ni ions would see a wider environ- 
ment with trigonal symmetry only, b) actually, oxygen 
octahedra are distorted^*^, and the Ni-O-Ni bond angle 
is Ri 96.4 ° in the case of Na, and w 94 ° in case of the Li 
compound. 

If there is a Jahn-Teller phase transition (as in 
NaNi02), it lowers the crystal symmetry further, and 
makes the orbital ground state unique. An alternative 
would be to ascribe orbital polarization to an electronic 
phase transition due to orbital exchange, and to regard 
the lattice distortion as an induced secondary effect. In 
what follows, we assume trigonal point group symmetry 
which is valid for NaNi02 at high temperatures, and for 
LiNi02 at all temperatures. Breaking the local trigonal 
symmetry, whenever it happens, is ascribed to orbital or- 



dering. We consider electronic degrees of freedom only, 
but we assume that the lattice would follow the changing 
orbital occupation. 

The Ni'^+ ions are in the 5=1/2 low-spin state. In 
terms of the dominant cubic component of the crystal 
field 3cf = ^2369- Since the actual point group symmetry 
is trigonal, t2g gets split into two levels {t2g A2 + E, 
where standard notations for the irreps of the point group 
I?3rf were introduced) but this docs not affect the fact that 
6 electrons are taken up by closed subshells, and only 
the seventh electron is in an open subshell. The trigonal 
crystal field component changes the detailed nature of the 
d-states, but still allows for twofold orbital degeneracy: 
gcubic _^ jTj^ what follows, E is understood to denote 

the two-dimensional irrep of the trigonal point group4. 

The ground state of an isolated Ni'^^ ion is fourfold 
degenerate: it has twofold orbital, and twofold spin de- 
generacy. A standard scenario would be that the non- 
Kramers degeneracy is resolved by a (cooperative) Jahn- 
Teller effect, while the Kramers degeneracy is lifted by 
magnetic ordering. Let us note that, as far as the E- 
electrons are concerned, the cooperative Jahn-Teller ef- 
fect is synonymous with orbital ordering, thus it can be 
explained with a purely electronic model, without the 
consideration of electron-lattice coupling. 

The standard scenario seems to be (at least nearly) 
realized for NaNi02, which has a first order cooperative 
Jahn-Teller transition lowering the local symmetry from 
trigonal to monoclinic at Ty^ ~ 480KS. The remaining 
Kramers degeneracy is lifted by a magnetic transition at 
Tj^^ « 20K, which was characterized as the antiferro- 
magnetic ordering of ferromagnetic planes^. In contrast, 
LiNi02 does not undergo a Jahn-Teller distortion^, and 
though the measured susceptibility shows a number of 
anomalies, it does not seem to develop magnetic long 
range orderi^iL^. It is puzzling that the two isostructural 
and isoelectronic compounds show so different behavior. 
Naturally, it is always possible that some of the observed 
behavior is not intrinsic. Impurities and structural de- 
fects are likely to prevent orbital ordering. Indeed, it 
was suggested that only NaNi02 allows the growth of 
sufficiently good-quality samples, and the observation of 
ordering transitions, while the overall behavior of LiNi02 
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FIG. 1: ANi02 structure. Ni ions are located in the middle 
of the O octahedra. 



is like that of the high-temperature phase of NaNiOa^. 



II. SPIN-ORBITAL MODEL BASED ON THE 
TRIGONAL DOUBLET 

The aim of the present paper is to show that the con- 
trasting features of NaNi02 and LiNi02 appear naturally 
as nearly equivalent possibilities for the intrinsic behavior 
of spin-orbital models of the trigonal E doublet. 

A similar four-state model, namely the 5=1/2 cubic 
eg doublet on the cubic lattice, has been studied in great 
detail in the context of manganite physics^. The mag- 
netic behaviour is complicated because orbital and spin- 
orbital interactions tend to frustrate the usual spin-spin 
interactions. Though, in contrast to spin-only models, 
spin-orbital models do not need the fine-tuning of the 
lattice structure to get frustration effects, we find that 
the geometrical frustration of the triangular planes of the 
LiNi02 structure brings essential new features. For this 
reason, we consider only an isolated triangular plane, and 
discuss T = behavior only. We assume that our essen- 
tial conclusions would carry over to the T > behavior 
of coupled planes. 

The idea that the geometrical frustration of the tri- 
angular lattice tends to oppose ordering, has been dis- 
cussed for spinifliii and orbitaliS degrees of freedom sep- 
arately. In a pioneering work, Hirakawa et al^ started 
a systematic investigation of triangular lattice antifer- 
romagncts with the explicit aim of finding non-Ncel- 
type behavior. This work initiated the intensive re- 
investigation of LiNi02. On the theoretical side, Arimori 
and Miyashita— studied a classical model and found that 
novel order parameters combining spin and orbital char- 
acter are important. In a quantum- mechanical calcula- 
tion, chosing a special set of parameters to make the four- 
state spin-orbital model SU(4) symmetrical, it was found 



that the ground state of the nearest-neighbour model 
on the triangular lattice is an SU(4)-resonating quantum 
liquidi^. 

Here we consider the full range of E models, restrict- 
ing the parameters only by the requirements dictated 
by symmetry. The pair interaction is generically of 
SU(2)(g)C2/i symmetry; higher symmetries (SU(2)®SU(2) 
or SU(4)) follow from specific choices of the parameters. 
We explore many lower-symmetry states in addition to 
the fully symmetrical SU(4) phase. Accepting that the 
observed behavior of both LiNi02 and NaNi02 is intrin- 
sic, any theory for why LiNi02 does not order should 
also allow for the alternative scenario of orbital and spin 
ordering, as observed in NaNi02. In terms of our trigo- 
nal E model, we show that the combination of geometri- 
cal frustration with the dynamical frustration inherent in 
spin-orbital models gives rise to a rich variety of compet- 
ing states stretching from the SU(4) resonating singlet 
state to spin-ferromagnetic phases with various orbital 
order. We will find it natural that contrasting behavior 
resembling that of either LiNi02 or NaNi02 can arise in 
nearby regions of parameter space. 



A. Basis functions 

Our model is meant to describe the Mott-localized E 
electrons of Ni ions. The local degrees of freedom are 
those of an E^ shell. Intersite interactions arise from the 
virtual charge fluctuations E^E^ E^E^. The study of 
such spin-orbital exchange models was initiated by Kugcl 
and Khomskiii^, and by Castellani et ali^. 

The point group of a Ni site is = V^^IS ,1}, where 
£ is the identity element, and X is the inversion. The 
subgroup of proper rotations contains the trigonal 
axis C3 and three orthogonal C2 axes. It is convenient to 
denote axes in terms of the original octahedral system 
{X,Y, Z}, so the C3 axis is (111). For later reference, we 
recall that V^d has three irreps: the identity rep Ai, the 
one-dim irrcp A2, and the two-dim irrcp E. 




FIG. 2: a. The C3 axis and one of the C2 axes of the point 
group of the Ni site in the ANi02 structure (the other two C2 
axes are obtained by applying C3). b. Orbital states of the 
seventh d-electron of Ni'^^ on the backgound of the network 
of oxygen octahedra. 



First, we represent on the basis of the E subspace 
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spanned by ct|0) = \a) oc {3Z'^-R'^), and = cl\0) = \b) cx 
{X'^ — Y'^) (as yet, we omit the spin index). Alternatively, 
we may represent on the two-dimensional operator sub- 
spaces {cj,,cjj} (or {ca,Cb}). The effect of a 7r-rotation 
about the 110 axis (one of the C2 axes) is 



Skipping the effect of the other two C2 rotations, we show 
how a 27r/3 rotation about the trigonal axis is represented 
in the E subspace 




B. The microscopic model 

On-site d-electron orbital states are classified according 
to the point group V^d-^, while two-site states according 
to the smaller point group C2h of a pair. The nearest 
neighbours of a Ni site are at the centers of octahedra 
which share an edge with the first site. The C2 axis per- 
pendicular to this edge is a symmetry element of the pair; 
so is the mirror plane ah perpendicular to the C2 axis in 
questioniS. C2 and ah are the generators of the 4-element 
symmetry group C2h of the pair—. 

The standard components of the electronic hamilto- 
nian are those of a two-band extended Hubbard model: 
intcrsitc hopping Tihop, and on-site Coulomb matrix ele- 
ments Hcou\- 

First, we discuss Hhop- The local E basis can always 
be chosen so that under the C2 rotation of the pair, one 
of the basis states is even, and the other is odd. In fact, 
we have seen this in This immediately implies that 
the hopping elements between two sites are only between 
the functions with equal parity, and we have two hopping 
parameters only: t for the "a" orbitals, and t' for the "b" 
orbitals 

Hhop ^-tYl 4,a,aCj,a,a - t' 4^b,a(^3A<y + H-C. (3) 
a a 

where a is the spin index. In the other directions the hop- 
ping amplitudes can be obtained by a suitable rotation of 
the basis functions and the hopping matrix. Let us note 
that for pairs with a different orientation, inter-orbital 
hopping terms will be generated. 

"a" and "b" need not mean strictly Ni d-states but 
rather more extended one-electron states of the same 
symmetry. Since one of the main pathways of electron 
propagation would be through the oxygen network, we 
should think of the orbitals as hybridized Ni-centered 
Wannier orbitals, but with hopping amplitudes which fol- 
low not only from Ni-O-Ni hybridization, but by consid- 
ering all finite-amplitude processes which are symmetry- 
allowed, and which in the end-effect may be indexed in 



the same way as the simple nearest-neighbour Ni-Ni hop- 
ping. 

It was noted by Mostovoy and Khomskiii that the as- 
sumption of exactly 90° Ni-O-Ni bond angle results in 
a peculiar form of the spin-orbital effective hamiltonian. 
In particular, spin-spin coupling is exclusively ferromag- 
netic, and orbital exchange predominates. One of the 
ways to look at the situation is that, with an ideal oc- 
tahedron of oxygen atoms, one-electron terms would not 
allow the propagation of an electron from a Ni site to 
another Ni site via an intervening oxygen atom. How- 
ever, other off-diagonal elements, like the spin flip part 
of the p-shell Hund coupling, still allow electron propaga- 
tion, and a corresponding term in spin-orbital exchangei. 
This model may be used to describe NaNi02, but it is cer- 
tainly not applicable to LiNi02. Since we aim at deriving 
both kinds of behavior from formally the same hamilto- 
nian, we have to pay particular attention to the sources 
of deviation from the Mostovoy-Khomskii scheme. 

Dare et al— pointed it out that the trigonal splitting 
of the oxygen p orbitals, and the deviation of the Ni- 
O-Ni bond angle from 90°, facilitate the appearance of 
antiferromagnetic Ni-Ni interactions. However, they did 
not systematically explore the phase diagram, and ne- 
glected several effects which we think are important: the 
direct overlap of the Ni wave functions at neighbouring 
sites, and the intra-atomic exchange and double hopping 
terms of the d-d interaction at Ni sitesS^. Our aim is 
a systematic investigation of the phase diagram in the 
entire parameter range. 

The form of the on-site pair interaction term TYcoui 
is restricted by the symmetry classification of the two- 
electron states: V^d for the orbital component of the 
wave function, and SU(2) for the spin part (which read- 
ily gives three singlets and a triplet). Orbital quan- 
tum numbers follow from E®E = Ai + A2 + E. The 
anti-symmetrical A2 can be taken with symmetrical spin 
states, yielding the triplet 

1^3) = ii4jo). (4) 

Ai and E are symmetrical, thus there must be two singlet 
levels: the non-degenerate Ai, and the two-fold degener- 
ate E. The E basis functions are 

\Fr.) = ^{cl,cl,-clcljO). (5) 
and the Ai basis function is 

l^6)-4s(iT<i+<T<i)|0)- (6) 
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Their transformation scheme under C3 



\Fi) 



\F4) 



V3, 



follows from ||2Jl. 

The most general on-site two-body Hamitonian de- 
scribing the E(E)E set of levels is 

Hcoui = ^n'^ - ./^(SaSb + pliant) 



where the U is the familiar on-site repulsion of the Hub- 
bard model, Jh is the Hund's coupling and Jp is the 
pair hopping amplitude. The spectrum of TYcoui con- 
sists of a triplet level at U — Jh i\Fi), IF2) and IF3)), 
a twofold degenerate singlet at U (IF4) and IF5)). and a 
non-degenerate singlet a,t U + 2Jp (l-Fe))- 

Since each of the single site terms is invariant under ro- 
tations in the orbital space, Ticoui written in Q is quite 
general, and its two independent parameters Jh/U and 
Jp/U could be chosen arbitrarily. We may think of these 
as effective interaction parameters which encompass all 
allowed processes affecting the E level under consider- 
ation. According to the usual evaluation of the simple 
Coulomb interaction we get Jp = Jh/'^- This physically 
motivated assumption was used by Castellani et al. in 
their pioneering worlJi on V2O3. See Ref.i for further 
discussions of this point. 



C. The effective hamiltonian from symmetry 
considerations 

The four-dimensional Hilbert space of E^ states sup- 
ports 15 local order parameters'^. Their standard choice 
is S"^, Sy, S"^ for the spins, T^, T^, for the or- 
bitals, and further nine operators S^T^, S^T^, ... of 
mixed spin-orbital character—. Here we introduced the 
T = 1/2 pseudospin operators 



rjiX 



a 

= i V f c -S c 



\,h.a i.a.a 



(8) 



For the present, we exploit the separation of spin and 
orbital Hilbert spaces, and do not discuss the mixed or- 
der parameters, though they are certain to be as relevant 
as S and T in high-symmetry situations. The symmetry 
classification of the orbital order parameters is obtained 
by representing the point group "D^d on the basis of the 



order parameters. In fact, since T^, T^, and are com- 
posed as c|jC/3, the representation we seek is the product 
of the representation Q)-© with its adjoint, and the de- 
composition E®E = Al + A2 + E can be used again. It 
turns out that and form the basis of the irrep E 
(a quadrupolar doublet), while transforms according 
to A2. We quote the transformation of the quadrupole 
operators under the C3 rotation 



a/3 



/O 1 

"2 2 



(9) 



From it is clear that 



CoTy 



-ry. and 



C2T' 



rj-iZ 



(10) 

Finally, let us mention that and Tz are time-reversal 
invariant. The fact that under the time reversal trans- 
formation T, TTx = Tx and TT^ ~ , shows that these 
arc quadrupolar order parameters. On the other hand, 
for the pure imaginary operator T**, TTy = —Ty. In 
the usual treatment of a cubic Cg doublet, would be 
an octupolar order parameter. However, under trigonal 
symmetry, A2 is also assigned to the dipolar order param- 
eter Liii (orbital angular momentum along the 111 direc- 
tion). Thus our must be a mixed dipolar-octupolar 
order parameter, but we will not analyze its nature in 
detail. 

The form of the effective pair interaction is restricted 
by the geometrical symmetries of the pair, and the nature 
of the order parameters ijHJ). We consider a pair of sites 
1 and 2 connected by the C2 axis which figured in our 
previous considerations. The other symmetry element is 
the perpendicular mirror plane Uh bisecting (1, 2). 

The orbital component of the lowest order effective 
hamiltonian consists of terms T^t!^ (a, /3 = x,y, z), and 
also of single-site terms like + T2 (reflecting that the 
choice of the basis is tied to this particular C2 axis). The 
pair energy expression must be invariant under C2, T, 
and also ah. cTh acts like 



^2 1 



GhTl - -r|, and GhTl 



rpZ 

^2 • 



(11) 

Time- reversal invariance excludes terms like rfTj, 
and also Tf + Ty, and either ^ or ^ exclude Tf+T^. 
In addition, (|10|l excludes also rfr|. Thus we are left 
with 

W'i2 = A,T^T^ + AyTfry + i,TfT| + A',{Tl + T^) . 

(12) 

where A^^ Ay, A^, and A'^ are some real coefficients. Let 
us emphasize that, in general, the coupling term TfTy 
may appear in the hamiltonian. 

Once we introduce spins in the problem, the same ar- 
guments hold as above with or without spin exchange, so 
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the Hamiltonian becomes 



which is more conveniently written as 



+ [B' + BT^T:^ + ByTfTy + B^Tf T| - 
+B',{Tl + T^)\S,S2. 



(13) 




6 7 

FIG. 3: The six neighbors of site 1. 



Now we consider bonds with different orientation. Lat- 
tice site 1 has six nearest neighbours 2, 3, 4, 5, 6, and 7, 
which form a regular hexagon (Fig. The interaction 
energy must be the same for the pairs (1,2), (1,3), etc., 
but just for this reason, the form of the pair hamiltoni- 
ans cannot be. They have to be derived from 7iy=i2 by 
suitable transformations. 

Consider first the (1,5) pair, which is the mirror image 
(either through a Uh plane containing 1, or by inversion 
through site 1) of the (1,2) pair. We can use (|ll|l to 



deduce Ti 



"^5 ' 



^2 



and 



T§, thus 



'Hij=i5 is of the same form as 7iij=i2- 

The (1,4) pair interaction can be deduced by the C3 
rotation [Eq. Q] from Eq. ifT^ . In fact. 



V3 



-r.f 



V3 



V3 1 

~ 2^1 



1 



Ti ^ ni4-Ti 



ni4-T4 



(15) 




with ni4 = 



and the column vector T = (T^, T?', T^). 



Similarly, riis = 



For pairs of different orientation (e.g., (1,3)), analo- 
gous expressions can be given. For the pair ij, the ef- 
fective hamiltonian Tiij could be deduced from by 
replacing T" — > (n^-T)" — nf T everywhere, where the 




n" with a 

'■J 



X, y, z denotes the first, second or third row 
of matrix riy , respectively. However, it is worth to note 
the following simplification. Under these transformations 
Tf T| TfT^, and T^T^ + TfTi T^Tf + T^T^ . In 
other words, the orbital base changing transformations 
have the character of a rotation about the pseudospace 
y axis. Pseudospin-space symmetry is easier to identify 
if in the first line of Eq. (|13|1 we make the following rear- 
rangement: 



AT^T^ + AyTyry 



A.TfT! 



AT.Tj + {Ay - A)Tl'Tf + {A, - A)T^T^ 



at:,t. 



A^fT]- 



A.TfTf 



(16) 



TiT,,- and Tj'T^ are invariant, so from the intersite in- 
teraction terms, only the coefficient of the T/TJ term 
depends on the orientation of the pair. The effective pair 
Hamiltonian is then: 



'Hi 



[AT,T 
+ \B' + BT,T 



,^AyTfT] + A,{ja:i^Ti){pl^j) 



ByTfT] 



B.(nf^-TO(n^,T,) 
I 



S) S 7" 



(17) 



For completeness, we list here nfj in all the six possible 
directions: 



The lattice hamiltonian 



(21) 



n?5 = (0,0,1) 



M3 



^16 



^17 




(18) 
(19) 

(20) 



is the sum of ()17|l over all nearest-neighbour pairs. In 
the summation the A'^ coefficient drops out as nf2Ti + 
nf^Ti + nfgTi = 0. The Hamiltonian above contains 
pure orbital couplings, pure Heisenberg spin exchange, 
and also terms of coupled spin-orbital character. The 
only symmetries of the lattice hamiltonian are global 
SU(2) for the spins, and the space group symmetry. 
We note here that our theory can be applied to any 
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triangular d} system whose local Hilbert space is the trig- 
onal doublet E. The structure of BaVSs can be envis- 
aged as the sequence of triangular planes of V''+ = 3d^ 
ions. It has been argued that even the minimal model of 
BaVSa should include the orbital degrees of freedomS^. 
If one assumes that the lowest-lying crystal field level is 
the E doublet derived from the trigonal splitting of t2g, 
our present considerations become relevant for BaVSa as 
well. 



We found it convenient to express the Hamiltonian using 
the Vij^^ and Vfj^^ projection operators onto the singlet 
and triplet spin combination on the bond: 

nf° = J-S,S, and V^=' = S,S, + ^ (23) 

First, some general remarks about the parameter range. 
(|7|) shows a two-parameter manifold of on-site Coulomb 
hamiltonians. However, we do not change Jp/ Jh con- 
tinuously, but investigate two special cases only: (a) ne- 
glecting pair hopping Jp = (a frequent, though not 
clearly motivated, simplification), and (b) the physically 
motivated choice Jp = Jh/'2- Most of our results will be 
about the latter case, using the notation J = 2Jp = Jh- 
Redefining the basis states (jja <-> 06 interchanges the 
definitions oft and i', thus it is sufficient to consider the 
\t\ > |t'| case. 

It is, however, worth noting that the orbital part of 
(I22|l becomes SU(2) invariant for t ^ t' and Jp ~ 0: 

n, . f (t,t, + 5) (s.s, - i) 

The lattice hamiltonian has now global SU(2) symme- 
try for the spins and global SU(2) symmetry for the 
pseudospins (global SU(2)®SU(2), with the six conserved 
quantities Y,j Sf, foi' a,l3 = x, y, z). 

A still higher symmetry is obtained for Jh ^ Jp = 
when the pair Hamiltonian simplifies to the SU(4) 
symmetrical^^ 

H,, = (^T,T, + [SS, + ■ (25) 



D. The effective hamiltonian from microscopic 
model 



Symmetry considerations do not allow to obtain rela- 
tionships between the A and B coefficients; they may be 
derived from the model lj2J) and Q by second-order large- 
U perturbation theory, as usual for Kugel-Khomskii 
hamiltoniansi^. As a result, we get 



,s=o 

ij 



(22) 

I 

The corresponding lattice hamiltonian possesses global 
SU(4) symmetry (there are fifteen conserved quantities: 
Sf, 7f , and SfT^ for a,p = x, y, z). 

III. GROUND STATES OF THE PAIR AND THE 
TETRAHEDRON PROBLEM 

In what follows, we seek to find the possible different 
types of ground state of H21|) on the triangular lattice. 
For a first orientation, we describe the results for small 
systems, then go over to larger ones. Whenever possible, 
we use preconception-free numerical methods, and then 
try to re-interpret the results with approximate theories 
which can, in principle, be generalized to infinite system 
size. It is a general trend that with increasing system size, 
complicated states are found whose existence could not 
have been guessed by simple-minded extrapolation from 
small systems. Therefore we will have to be cautious in 
drawing conclusions about the thermodynamic limit. 

Before we turn to the physically motivated Jp = JhI"^ 
case, we examine the case when the pair hopping ampli- 
tude is absent. 



A. Two site problem 

For simple spin models, the correlations found for a 
pair of sites allow to infer the character of the ordered 
phase in the thermodynamic limitS&. Our first aim is to 
map the pair solutions, and try to deduce how spin and 
orbital order may complement each other. 

The most notable consequence of setting Jp = is that 

the Att'T}'T^ term cancels from the first and second row 



" the effective Hamiltonian (|22|l . Naturally, there is still 



U+2J, 
2 



1 



(t^-i'^)(<T, + n^,T,) + ^(t^+<-) 



1 



U 



Att'TfTf 



1 



V 



_(i^+t-) + _(i^_i-)(n^T,+n^T,) 



,5=0 



U-Jh 



-2tt'T,T, -{t- f')'(nf,T,;)(n^T,) + -{t^ + t'^) 



,s=i 
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a T^Tj interaction included in the isotropic term T^Tj. 
On this basis, one may not expect a preference for T^- 
polarized (complex) orbital ground states. However, one 
should not overlook the possibility that the system may 



choose T'^-polarization as a compromise when interac- 
tion terms preferring real orbital order mutually frustrate 
each othei"2i. 

Let us note that the hamiltonian of the ij ^ 12 bond 



2 

U 



2tt' {T[T^ + TfTl) + {e + f'2)Tj^T| + {f - t'^){Tl + r|) + ^{t^ + t'^) 



U-Jh 



■pS=l 
' 12 



(26) 



has two new symmetries characteristic of the two-site 
problem. One of them is axial symmetry about in 
pseudospin spacers, which allows to classify the eigen- 
states as TI + T2 eigenstates. The other is a the t' —t' 
symmetry: a 7r-rotation about in pseudospin space for 
site 2 is a canonical transformation which leaves the en- 
ergy unchanged, but it amounts to t' —^ —t'. This sym- 
metry can be restated for larger clusters with bipartite 
structure, but it cannot be extended to > 2 clusters 
of the triangular lattice. 

For t = t' SU(2)(XiSU(2) symmetry follows as in (|^ . 
Taken in conjunction with the previous remarks, the t = 
—t' model must have the same symmetry. Similarly, the 
degeneracies must be the same for the SU(4) point t = t' , 
Jh = 0, and its mirror image t = —t', Jh = 0. 

The Hilbert space of two electrons on two sites is 16 
dimensional, and the energies and orbital eigenstates for 
the ij = 12 bond are: 

Es^o^-^' ^5=1 = 0, \aa) 

i?s=o = -^, Es=i^O, \bb) 

Es=o = -—jj-^, Es=i = - fj_j , \ab) + \ba) 

Es=o^-^, Es=i^~^, \ab)-\ba) 

The results are shown in Fig. 0] Inside any of the 
ground state phases, cither the spins are parallel, and 
the orbitals antiparallel, or vice versa. At the boundaries, 
the ground state level has higher degeneracy which can 
be interpreted as the manifestation of one of the higher 
symmetries discussed above. 

For the spin singlets, t ^ t' acts like an external or- 
bital field, and therefore orbital polarization is either in 
the a, or the b direction. At the line t = t' (section 
Jh < 0) not only aa and bb (states from the neighbour- 
ing domains) become degenerate but also \ab) + |5a), thus 
the ground state is the threefold degenerate spin-singlet- 
orbital-triplet (5 = 0, T = 1), allowed by the symmetry 
SU(2)®SU(2). Continuing the t = t' hnaSa to Jh > 0, 
the ground state is again threefold degenerate, but its na- 
ture changed to spin-triplct-orbital-singlet {\ab) — \ba)). 
The border point t ^ t' , = is the SU(4) point where 
the (5 = 0, T = 1), and (5 = 1, T = 0) levels become 
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FIG. 4: Zero temperature phase diagram of the model on 
two sites as a function of t' /t and Jh/U- We have indicated 
the spin and orbital parts. Along the thick line at t' /t = 1 
and Jh < the ground state is triply degenerate: the orbital 
part becomes SU(2) symmetric, and the orbital triplet \aa), 
\ab} + 1 6a) and \bb) with the spin singlet forms the ground 
state wave function. Along the t' /t = — 1 and Jh < line, 
the orbital triplet consists of \aa), \ab) — \ba} and \bb). At the 
SU(4) points the ground state is 6-fold degenerate. 



degenerate [forming the basis of the six-dimensional anti- 
symmetrical irrep of SU(4)]. 

Analogous results hold for the other SU(2)(g)SU(2) hne 
t — ~t' . only for Jh > 0, the \ab) — \ba) orbital state (T = 
0, = 0) is changed to \ab) + \ba) {T = 1, 0). Note 

that the energy difference between these two states comes 
from the term of ^ proportional to t'iT^T^ +T^T^), 
so changing the sign of t' changes the parity of the ground 
state. The sixfold degeneracy of the ground state at the 
"anti-SU(4)" point t = -t' , Jh = follows from the 
t' <-^- —t' symmetry of the pair problem. 

A very similar phase diagram would be obtained for 
Jp ^ 0. We do not show it here, but we include the 
contribution of pair hopping in all our subsequent calcu- 
lations. 

If any far-reaching conclusions from Fig. ^ could be 
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drawn, it would be that the ground state has either 
ferro-orbital order and spin antiferromagnetism (or a sin- 
glet spin liquid); or it is a spin ferromagnet with stag- 
gered orbital order (or orbital liquid). Less obviously, at 
the SU(4) points, a spin-orbital quantum liquid may be 
inferredi^. 

To either confirm, or disprove, these guesses, two 
routes can be followed: a) determine the exact phase 
diagram of larger clusters and see if there is a clear trend 
emerging; b) construct variational wave functions which 
possess the envisaged correlations. Fig. 0] suggests that 
antiferromagnetic effective spin models can be derived 
easily because uniform orbital order factorizes site-by- 
site. However, for high-spin states, the orbital states are 
more complicated. SU(4)-like states, for which spins and 
orbitals are entangled, pose further challenge. 

B. Four site problem 

In what follows, we set J = 2Jp = Ju- The four site 
cluster with periodic boundary conditions is equivalent to 
a tetrahedron where the three directions on the triangular 
lattice correspond to the three pairs of opposite bonds 
on the tetrahedron. This cluster proves to be sufRciently 
large to provide us with some insight into the problem. 

As a first step, we do exact (numerical) diagonalization 
for the hamiltonian 

■^tetr = ^12 + Wl3 + 7Yi4 + Ti.23 + ^24 + ^34 (28) 

in the 4'^ = 256-dimensional Hilbert space. The total spin 
S and its z-component 5^ are good quantum numbers, 
but this is only of limited use in identifying eigenstates. 
The hamiltonian couples spin correlations with orbital 
correlations, therefore most of the eigenstates have mixed 
spin-orbital character. More precisely: the S = 2 eigen- 
states can be sought in the factorized form 

I tTTT>®$(ri,T2,T3,r4) 

but this is no longer true of lower-spin states. In partic- 
ular, we know that there are only two independent spin 
singlet states, [12] [34] and [23] [41], but combined with 
the orbitals, we have 32 independent S — spin-orbital 
states. Most of the = eigenstates of Eq. H28(l are not 
represented as a linear combination of the above singlets 
multiplied by a pure orbital state. In fact, an overall sin- 
glet which plays a prominent role in our considerations 
is the SU(4) plaquette singlet 

vE'su(4) = [12]{23}[34]{41} - [23]{34}[41]{12} (29) 

where {23} represents the pseudospin singlet connecting 
sites 2 and 3, etc. It is clear that \E'su(4) is a spin singlet, 
just as it is a pseudospin singlet, and it does not factorize 
in spin and orbital variables. ^"311(4) is the ground state 
of Titotr in the SU(4)-symmetrical point {t = t' , J = 0) oi 
the parameter space of (|28|) . It is the only SU(4) singlet 



(the only basis function for the 1-dim irrep of SU(4)) in 
the present Hilbert space. 

We diagonalized H28|l . and followed the low- lying states 
(a representative example is shown in Fig. [S]) . While the 
detailed nature of the ground state always has some con- 
tinuous dependence on the hamiltonian parameters t'/t 
and J/U, there are also sharp changes at level cross- 
ings. A level crossing is possible between states with 
different symmetry labels. The symmetry of Titctr is 
SlJ{2)(E)Td, where Td is the tetrahedral group. Td has 
one- and three-dimensional irreps; furthermore, granting 
time-reversal invariance, two complex conjugate one-dim 
irreps belong to degenerate energy levels. We distinguish 
between ground states according to their spin degeneracy 
(25" -I- 1), and orbital degeneracy which can be 1, 2, or 3. 




-1 -0.75 -0.5 -0.25 0.25 0.5 0.75 1 



t'/t 

FIG. 5: Energy spectrum of the spin singlet sector in the 
tetrahedron as a function of t'/t for the Jh ~ Jp = case. 
For t'/t between 0.2 and 1 the ground state is well separated 
from the rest of the states and it is the adiabatic continuation 
of the SU(4) singlet state at the t'/t — 1 point (full line). 
The SU(4) singlet nature of the ground state is lost at around 
t'/t = 0.2 (denoted by an arrow), and in the region —0.7 < 
t'/t < 0.2 three levels (a non-degenerate level (solid line) and 
a 2-fold degenerate level (dashed line)) go together. Finally, 
at t'/t = —0.7 the symmetry of the ground state changes, 
indicating the appearance of the third phase (dashed-dotted 
line). 



The resulting phase diagram is shown in Fig.|Sl Phase 
boundaries were drawn where we found a clear change in 
the character of the ground state; this holds also for the 
boundary between the two non-degenerate (S = 0, Ix) 
phases. Let us immediately point out that the tetrahe- 
dral phase diagram is very different, and therefore would 
have been difficult to guess, from the pair phase diagram 
shown in Fig. 0] Taken in itself, the lack of mirror sym- 
metry about the <' = axis was to be expected, since 
the tetrahedral cluster is not bi-partite. In particular, 
the t' = —t, Jh = point does not have any special sig- 
nificance. However, less obvious features are the variety 
of singlet phases, the shrinking of the domain of spin- 
polarized solutions, and the predominance of the singlet 
phase into which the SU(4) point is embedded. 
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FIG. 6: Phase diagram of the spin-orbital model with 
J = Jh ~ 2Jp and U = (7 — J on a tetrahedron, based 
on exact diagonalization (see also Fig. |^ . Phase boundaries 
in bold lines belong to level crossings in the ground state en- 
ergy. A further singlet-to-singlet transition is identified in the 
vicinity of an antilevel crossing (dashed line) . The degeneracy 
(apart from the trivial spin degeneracy) of each state is also 
indicated. 



IV. VARIATIONAL APPROACH FOR THE 
FOUR SITE CLUSTER 




FIG. 7: The phase diagram of the effective model on a tetra- 
hedron based on the spin-orbital decoupling scheme (1301 . 



A. The method 

The previous section showed us that we could expect 
a rich phase diagram for our model even on a small size 
cluster. Wc will continue our investigation by studying 
larger clusters by a kind of variational method: since 
there is a strong asymmetry between the spin and the 
orbital parts in the Hamiltonian, we try to decouple spin 
and orbital degrees of freedom^ by factorizing the wave 
function into a j^'"^) spin and IvP^) orbital part: 

|*st) = 1*^) ® 1*^) . (30) 



While this factorization applies to the pair problem, it 
cannot describe the entanglement of spin and orbital fluc- 
tuations for > 4 sites. In particular, it does not allow 
to capture the SU(4) character displayed by (|29|l . How- 
ever, it should work well for states with weakly fluctuat- 
ing orbital order. 

We proceed as follows: in the effective Hamiltonian 
we can separate a spin-orbital mixing term, and purely 
orbital terms: 

= 5]{2(Si.Sj)hJ + kJ} (31) 

Next, we need to minimize the Hamiltonian by using the 
factorizcd wave function: (^I^sTl-ff I^st)- It implies that 
l^'s) is an cigcnstate of the Hamiltonian 

^2(Si.Sj)(*T|hJ|*T) (32) 

while I'I't) is an eigenstate of the Hamiltonian 

^(2(vI/s|S,S,|vI/s)/iS+fc,5) (33) 

This coupled set of equations is solved by iteration, keep- 
ing at each step the solutions with lowest eigenvalue. 

We have applied this technique to obtain the phase 
diagram of the regular 4-sitc (tetrahedron) and 16-site 
cluster. 



B. The phase diagram on a tetrahedron 

As we can see in Fig.[7| the "mean-field" phase diagram 
shows a remarkable resemblance to the exact one (Fig-EJ. 
We should, however, note that the ferromagnetic region 
extended too much at the expense of the SU(4) phase, 
basically because the Ansatz (|3()|l cannot describe SU(4) 
correlations^, while the spin-aligned states are treated 
correctly. The S = 1 region has shrunk, too. Our vari- 
ational recipe for S ~ 1 states is to compose them of 
two bonds: a spin triplet and orbital \ab) + \ba) bond, 
and a spin singlet and orbital \aa) type bond. These 
can be permuted and rotated to give 6 solutions which 
are degenerate at the mean-field level. Allowing for the 
resonance between these six states, we can reproduce the 
3-fold degenerate 5 = 1 state seen in exact-digonalization 
study by taking the appropriate linear combinations of 
them. 

In the singlet sector wc can distinguish between sev- 
eral phases: The lowest one is composed of spin triplet 
and orbital \ab) + |&a)-like bonds, which are composed 
into a singlet, and is 3- fold degenerate at the mean-field 
level due to possible rotations (here again, the off diag- 
onal matrix elements between the states will favor the 
2-fold degenerate linear combination, in agreement with 
Fig.|SJ). In the remaining part, the spin wave functions is 
the same (singlet valence bonds), only the orbital charac- 
ter changes from \aa) type bonds to a more complicated 
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one close to the t = t' , J = SU(4)-symmetric point 
(where the approach we use is clearly not applicable). 
The number of solutions of the iteration becomes very 
large in the vicinity of the SU(4) point, with essentially 
the same energy. 

V. VARIATIONAL APPROACH FOR THE 
16-SITE CLUSTER 

The tetrahedron solutions show that there must be 
quite a few phases with markedly different spin-orbital 
correlations. However, the = 4 cluster is too small to 
draw inferences about the character of any emerging long 
range order (except for spin fcrromagnctism). Therefore 
we investigated an = 16 cluster which is large enough 
to differentiate between quasi-one-dimensional (chains) 
and genuinely two-dimensional orbital ordering patterns. 
We use the same variational method as for = 4. 

As shown in Fig. |S1 the model leads to a rich phase 
diagram. For reasonably large values of J/U we find the 
fully polarized ferromagnetic (S = 8) region with three 
phases that differ by their orbital structure. In the spin 
singlet region we can again distinguish at least 6 phases, 
which are labeled by capital letters. A detailed discussion 
of these phases follows. 
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FIG. 8: Mean-field phase diagram on a 16-site cluster as a 
function of hopping integral versus Hund's coupling. The 
grey phase is the ferromagnetic phase, with the classical phase 
boundaries shown (see section IV Bl 



A. The singlet phases 

The S = part of the diagram is composed of 6 phases. 
We have investigated in more details each phase starting 
from t' /t ^ —1 and going through the 4 boundaries until 
t' jt ^ 1 for several ratios J jU . The aim of this section is 
to understand the different types of orbital and spin or- 
ders. The nature of the spin phases turned out to be eas- 
ily determined from the variational method itself: In all 
cases except phase A, some clear pattern with large and 
positive or negative values of (Si.Sj) could be identified. 




FIG. 9: Spin and orbital structure in the singlet phases of the 
mean-field phase diagram. Solid line indicates AF, dashed 
line FM spin correlations. 



leading to magnetic or singlet dimer order. The orbital 
part was more tricky to identify since the most relevant 
operator is not Ti.Tj but /i^, and a given mean- value of 
this parameter docs not obviously lead to an orbital state 
since this operator is quite involved. So to get a simple 
physical picture of the orbital structure we have tried in 
each case to reproduce the pattern given by the mean- 
field solution for (ft-ij) assuming at each site an orbital 
wave-function of the form 

l^f) = (cos0.,|a,) +sina,|6,)) (34) 

and we have checked that this orbital structure also re- 
produces satisfactorily the mean value of Ti.Tj measured 
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in the mean-field ground state. This turned out to give a 
clear picture in all phases except A and E. The informa- 
tion obtained in this way is summarized in Fig.lHl In the 
following, we describe in more details all these phases. 



1. Phase A 

This phase contains the SU(4) point (t' = t, J = 0) for 
which the mean-field decoupling used here is known to be 
inadequate given the very symmetric roles played by the 
spin and orbital degrees of freedom^. In fact, it is be- 
lieved that at the SU(4) point the system is in a spin and 
orbital liquid state involving resonances between SU(4) 
singlet plaquettes. A discussion of the physical properties 
at the SU(4) point can be found in Ref. Although 
the mean-field solution is not directly relevant for that 
phase, the mean-field approach is still useful to determine 
the boundary of the SU(4) region since it allows to detect 
the domain of stability of the neighbouring phases, for 
which the mean-field solution is indeed relevant, as will 
be discussed below. As anticipated, the SU(4) physics ex- 
tends to a finite and relatively large portion of the phase 
diagram, and it can in principle be relevant for real sys- 
tems. Since our mean-field approach does not lead to any 
physical insight beyond the determination of the bound- 
ary of this phase however, we will not discuss it further 
here. 



2. Phase B 

From the magnetic point of view, this phase consists 
essentially of weakly coupled, antiferromagnetic chains 
(see Fig. EJ , while the orbital structure turns out to be 
rather subtle with an antiferro-orbital arrangement of 
ferro-orbital chains with orbitals which are neither pure 
\a) = \d-^^2_j.2) nor \h) = \dxi_yi) but alternate between 
^(|a) -I- 16)) and ^(-|a> + l^))- The detailed magnetic 
structure depends a priori on the residual couplings be- 
tween the chains. If the couplings arc equal in both resid- 
ual directions, some canting will presumably develop in- 
side the chains to accomodate the frustration, like in the 
limiting case of the 180 degree classical ground state of 
the Heisenberg model on the triangular lattice. This ef- 
fective magnetic hamiltonian would be similar to that 
realized in CS2CUCI4, with possibly spinon excitations 
as reported by Coldea et alS. If however the symmetry 
is broken between the residual directions, the system is 
expected to develop rather coUincar order, with lines of 
parallel spins along the direction of the most ferromag- 
netic or least antiferromagnetic residual coupling. For 
all parameters, the residual couplings predicted by the 
mean-field solution are very small, but their sign and 
symmetry depends on the parameters. They tend to be 
AF for small J and ferromagnetic for large J, and the 
symmetry between the two directions may or may not be 



broken depending on the parameters. While this interest- 
ing point would deserve further investigation, we do not 
think that a reliable answer to such a subtle issue can be 
obtained just on the basis of this mean-field decoupling, 
and we do not discuss the point further. 

3. Phases C and C 

Both phases are characterized by strong dimer singlets 
forming different regular dimer coverings of the triangu- 
lar lattice. On each dimer the orbitals are parallel, and 
they correspond to d^y2_j.2^ d^x^-^^ or d^y2_j.2 depend- 
ing on the orientation of the bond. Note that all these 
orbitals are Jahn- Teller active, leading in all cases to two 
long bonds and four short bonds. One might be tempted 
to conclude that these phases correspond to two types of 
valence bond solids with the patterns depicted in Fig. |51 
The mean-field approach has a very remarkable property 
however: In addition to the mean-field solutions with 
lowest energy shown in Fig. |3 there are several other 
mean-field solutions of the self-consistent equations with 
energies very close to the lowest energy corresponding to 
other dimer coverings of the triangular lattice. In such 
circumstances, going beyond mean-field is likely to cou- 
ple these solutions, and the relevant model would then be 
a quantum dimer model describing resonances between 
these states. As wc shall see below, this point of view is 
favoured by exact diagonalizations of finite clusters. So 
at that stage we think it is safer to think of these phases 
as a region of parameters where all dimer coverings are 
relevant states for low-energy physics. 

4. Phase D 

This phase consists essentially of weakly coupled an- 
tiferromagnetic chains, but in contrast to Phase B, the 
orbital structure is now ferro-orbital with only orbital 
d^z2_j.2, d^x2_j.2 or d^y2_j.2 depending on the overall di- 
rection of the AF chains. Since these orbitals are Jahn- 
Teller active, one expects in this case that the system 
would undergo a cooperative Jahn- Teller distortion with 
two long bonds per octahedra all pointing in the same di- 
rection. Like in Phase B, the actual magnetic structure 
will be controlled by the residual couplings, and all the 
discussion of Phase B applies here, including the sign of 
the residual couplings and the symmetry of the couplings 
in the directions of weak coupling. In that case too, a re- 
liable determination of the possible magnetic phases re- 
quires further investigation that goes beyond the present 
mean-field calculation. 



5. Phase E 

This phase is dominated by strong antiferromagnetic 
correlations in two directions and weak ferromagnetic 
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correlations in the third direction, leading to an effec- 
tive Neel structure. The orbital structure cannot be re- 
produced satisfactorily with the variational ansatz of one 
orbital wavefunction per site. The pattern of (hjj) would 
be consistent with a ferro-orbital ordering with orbitals 
-^(|a) + \b)) at all sites, but the Ti.Tj correlations are 
not ferromagnetic. So to decide on a possible orbital or- 
der would require to go beyond the present mean-field 
approach. 

B. The ferromagnetic phase 

In the ferromagnetic region the "P^j^^ spin singlet pro- 
jection is 0; so that the effective Hamiltonian (|22|l is 
reduced to the following form (neglecting the constant 
term) : 

HT - E [(^ " t')'KT,)(n^T,) + 2tt'T.T,] 

U — J fj - ■ 

(35) 

and the orbital structure only depends on the ratio t' /t. 
As shown in the phase diagram|Hlwe can distinguish three 
phases going from i'/t = — 1 to t' /t = 1. All the identified 
phases identified arc orbitally ordered phases. They can 
be understood starting from the classical limit, which in 
our case is equivalent to minimizing the energy of the 

I*) = n (cos^jla) + e'-^^ sm9j\b)) (36) 

3 

site-factorized wave function. The phase boundaries 
shown in Fig. |H1 are obtained by equating classical en- 
ergies obtained from the wave function of Eq. 1361 

1. Phase Fl 

For t' = t the Hamiltonian of the orbitals becomes 
the standard SU(2) symmetric Heisenberg Hamiltonian 
with antiferromagnetic exchange. In this case a three 
sublattice long-range order for the T pseudospins devel- 
ops. Away from the SU(2) symmetric point, the three 
sublattice LRO is stable up to t' = t/3, with the 120° 
configuration restricted in the (T^, T^) plane [in Eq. H36|) 
we choose 6 for Oj^s in the first, 6 + 2tt/3 for 0j's in the 
second, and 9 — 2tt/3 for ^^'s in the third sublattice, with 
= everywhere] with energy 

Eafo _ {t + ty_ 

We have shown a possible 120 ° orbital pattern with 9^0 
in the top of the Fig. ^] While the classical approach 
does not allow us to fix the value of 9, this degeneracy is 
probably lifted by quantum fiuctuations. 

For finite systems, the signature of the developing LRO 
can be found in the energy spectrum in the form of the 




FIG. 10: Schematic representation of the orbital orderings in 
the spin ferromagnetic case. 



Anderson's tower, as has been confirmed by Bernu et 
al. for the isotropic triangular lattica^i. These low ly- 
ing states (Fl, Kl and r4) can also be seen in Fig. 1111 
and they can be continuously followed up to the isotropic 
point t' = t, where they become the lowest lying pseu- 
dospin triplet excitations. Further evidence comes from 
the nearest- and next-nearest neighbour (T^Tj) correla- 
tions. There is a strong ferro-orbital correlation between 
a site and its second nearest-neighbours, e.g. (TjTj) ~ 
0.19 for t'/t = 0.8. 

2. Phase F2 

To understand this phase, we start from the <' = 
case, where the Hamiltonian H35(l is proportional 
to ^X^bonds (j^y Ti) (n^Tj), which can conveniently be 
transformed to 

E ([n^(T.+T,)]^-(n^T.)'-KT,)^) 

bonds 

bonds i 

= E K-(T. + T,)]%3Em')'-3A^r(r+i) 

bonds i 

At the classical level, the two squares can be minimized 
by choosing the T vector in the (T^,T^) plane so that 
on a given bond either = —Tj, or T.^ -I- is per- 
pendicular to n| . These conditions are satisfied with 
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the collinear orbital order shown in Fig. ^| wc choose 
|a) + |6) along every second chain with the bond variable 
nf^ — (0,0, 1), and \a) — \b) along the remaining chains 
(the orbital configuration is the same as in phase B in 
Fig. I^J. There are 6 such configurations, which can be 
obtained by translations and rotations, with variational 
energy 

EcL 1 3i2 - 2tt' + 3f 2 

— --i U-Jn ^ ^ 

The classical phase boundaries for this state are t' /t = 
-1/3 and t'/t = 1/3. 

In a finite system with periodic boundary conditions 
respecting the point group of the trianguar lattice, 
the linear combination of the 6 states will produce a 3- 
fold degenerate state at the M point in the Brillouin zone 
(state Ml in Fig. Ill|l . and 3 states at the F point, one 
non-degenerate and one 2-fold degenerate (Fl, and F3 in 
Fig. El respectively). These states can clearly be recog- 
nized in the exact diagonalization spectrum of the 12 site 
cluster as the lowest lying state for —0.2t < t' < 0.35i, 
well separated from the states with higher energy. The 
observation of the phase in the correlation function is 
non-trivial, as the ground state around t' — is twofold 
degenerate, and the applied exact diagonalization on a 
finite size cluster will result in a state with an arbitrary 
linear combination of them, which leads to a pattern dif- 
ficult to interpret. It is, however, clear that there is no 
ferro-orbital order. 



3. Phase F3 
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FIG. 11: Energy level scheme of a 12 site diamond-like clus- 
ter with periodic boundary conditions and compatible with 
the 3-sublattice LRO. Shown are the levels which can be as- 
sociated with the ferro-orbital T"^ order (denoted by Fl and 
r2), collinear phase (Fl, Ml, and FS) and the lowest states 
constituting the Anderson tower of the 120 ° antiferro-orbital 
phase (Fl, and F4). The first letter refers to the momen- 
tum of the state. We have encircled the level crossings which 
we used to determine the phase boundary (t'/t = —0.20 and 
t7t = 0.35). 



phase boundaries by level crossings of the 'ground-state 
manifolds' associated with each type of ordering, which 
agree reasonably well with the classical phase boundaries 
(i! jt = ±1/3). At these phase boundaries continuous de- 
generacies appear in the classical wave function, suggest- 
ing a gapless excitation spectrum at those points. 



In this phase the ferro-orbital order is established: 
for negative t' the T^T^ term in Eq. H35|) becomes fer- 
romagnetic, and the frustration in the and due 
to the (nJ|^Ti)(n|jTj) term will single out the order. 
The particularity of the ordering is that it breaks 
the time-reversal symmetry: either the |a) -I- or the 
|a) — i|6) combination orders. The ordering of complex or- 
bitals has been searched for in the context of manganites, 
where it has been thought that they are favoured by the 
isotropic kinetic exchange. Indeed, the charge density of 
the I a) ± i I &) shows the trigonal symmetry, and the com- 
bination is Jahn- Teller inactive. The phase can be easily 
identified in the finite size diagolization from the correla- 
tion function: spatially isotropic TfT^ > correlations 
are dominant. The mean field variational energy of the 
ferro-orbital complex state is 



£^FO 



U-Jh 



(39) 



and the phase is stable for <'/t < —1/3. 

The determination of the phase boundaries is, however, 
not straightforward. As can be seen from the energy lev- 
els, the Fl state is present in the 'ground-state manifold' 
of all the ordered phases. Therefore we identified the 



VI. EXACT DIAGONALIZATIONS 

Due to the small number of conveniently exploitable 
symmetries in the problem (we have only the spin SU(2) 
symmetry), the size of the Hilbert space grows very 
rapidly with the size. In the S*^ = sector it increases like 
(^2)2^ where N is the number of sites. This limits us 
to small cluster sizes, especially if we want to explore the 
phase diagram. The obvious choice was the 12-site clus- 
ter with periodic boundary conditions (Fig. I12|l . which 
has the full point T>3d symmetry of the lattice as well. 
The considered cluster has the advantage to allow the 
formation of SU(4) plaquettes, and is also compatible 
with three and four sublattice order. 

The phase diagram obtained from the level crossings 
in the ground state is shown in Fig. E| It is globally 
consistent with the mean-field one. The fully polarized 
ferromagnetic region (S = 6) is found for very similar 
values of J/U. For small J/U, we identify 5 different 
regimes from t'/t ~ —1 to t'/t = +1. They seem to 
correspond to 4 phases only since two regions join for 
intermediate values of J/U, but given the difficulty to 
determine phase boundaries from exact diagonalizations, 
this should not be taken too seriously. The various re- 
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FIG. 12: The 12-site cluster with periodic boundary condi- 
tions, and the associated Brillouin zone 



gions are labelled according to the point in the Brillouin 
zone where the ground state is found. 



SU(4) 




0.05 0.1 



0.15 0.2 0.25 0.3 

J/U 



FIG. 13: Exact diagonalizations : phase diagram for the 
12-site cluster 



In the vicinity of the SU(4)-point {f /t ^ 1, J ^ 0), the 
low-lying spectrum is similar to the one obtained in the 
SU(4) casoi^. This suggests that the description of the 
ground state in terms of SU(4) singlets is applicable in 
this region. 

At the M point the ground state is three-fold degener- 
ate. This could correspond to the formation of AF chains 
in phase B since these chains can form in three direc- 
tions, resulting in a 3-fold degenerate mean-field solution. 
To confirm this interpretation, we have diagonalized the 
full Hamiltonian in the variational sub-space spanned by 
the mean- field ground state wave -functions of Phase B. 
It turns out that these states are not coupled because, 
due to the orbital configuration, they have different sym- 



metries with respect to the inversion around the middle 
points of nearest-neighbour bonds. So the ground-state 
degeneracy in this variational subspace is still equal to 3, 
supporting the interpretation in terms of chains. 

When the ground-state is at the K point, the inter- 
pretation is not so straightforward. The ground state 
is strictly speaking twofold degenerate. But looking at 
the spectra the first excited state is at the F point, 
and very close to the ground state. A possible expla- 
nation could be that all these states are degenerate in 
the thermodynamic limit. Then this region could also be 
explained by the formation of chains. To check this point, 
we have diagonalized the Hamiltonian in the variational 
sub-space spanned by the three mean-field ground state 
wave -functions of Phase D. The orbital configuration is 
different from Phase B, and the degeneracy is partially 
lifted, with a two-fold degenerate ground-state an a non- 
degenerate excited state. Again the agreement supports 
the interpretation of this phase in terms of AF chains. 

The most interesting region for our case is the central 
one. We will focus our attention on the line t' /t = 0. 
Along this line we will see that a description in terms 
of RVB states is reasonable. For instance, the low-lying 
spectrum (Fig. [HI) for J/U = 0.008 and t' /t = 0, shows 
a very large number of singlets states (125) before the 
first triplet (at the top of the figure). All these singlets 
are very close in energy, the energy difference between 
the ground state and the first triplet being of the order 
of ~ |j . Note that the number of singlets below the first 
triplet (125) is a significant fraction of the total number 
of dimer-coverings for this 12-site cluster (348). This is 
reminiscent of the spectrum found by Lecheminant et al 
for the S=l/2 Heisenberg model on the kagome lattice^. 
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FIG. 14: Low-lying singlets for t' /t = and J/U = 0.008 



This is in qualitative agreement with the mean-field 
results. Indeed, in phases C and C, several solutions 
corresponding to various dimer coverings were found with 
comparable energies (see Fig. I15|l . A similar observation 
was made in a preliminary study of a similar spin-orbital 
model in the context of BaVS^^^. 

A possible ground state for this region could be a spin- 
orbital version of the resonating valence bond (RVB) 
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stataiS. The magnetic structure could be envisaged as 
a fluctuating pattern of bonds among different dimer- 
coverings or a mixture between dimer-coverings and 
chains. All these states being singlets, they may be de- 
generate in the thermodynamic limit. 

Let us also mention that there is also a partially polar- 
ized region 5 = 3. Wc suspect that it may be a finite-size 
effect, as it is greatly reduced with respect to the corre- 
sponding S ~ 1 phase present in the phase diagram of 
the tetrahedron (Fig.|HJl. 




Energy per site = -0.188077 Energy per site = -0.186959 



FIG. 15: Two stable states for a 12-site cluster ; the dashed 
line represents the cluster, t' /t — and J/U = 0.008 




Energy per site = -0. 1 87823 Energy per site = -0. 1 88396 



FIG. 16: Two stable states for another 12-site cluster : the 
dashed line represents the cluster, t' /t = and J/U = 0.008 



VII. EXPERIMENTAL IMPLICATIONS 

In transition metal oxides, the on-site Coulomb repul- 
sion U is typically in the range 4-10 eV, and the Hund's 
rule coupling in the range 0.5-1 eV, leading to a physi- 
cal range defined by 0.05 < J/U < 0.25. Interestingly 
enough, all phases appear in this range and should be 
possible to observe in actual compounds provided the ra- 
tio t' /t has the appropriate value. In that respect, one 
should emphasize that the phase diagram depends only 
on the hopping integrals between orbitals, not on the ac- 
tual orbitals. In particular, even if the two orbitals were 
not orthogonal by symmetry on one of the bonds, di- 
agonalizing the hopping matrix on a given bond would 
bring us back to the situation treated in this paper. So 
the discussion would carry over beyond the specific case 
of (i322_^2 and dx2-y2 up to the remarks dealing with 
Jahn- Teller distortions. 



Now, coming back to LiNi02 and NaNi02, hence to 
d^z^-r^ and dx2-y2 orbitals, simple arguments suggest 
that f /t is negative and small. That it is negative comes 
from the different symmetries of the orbitals: the d322_r2 
orbitals on edge-sharing octahedra (see Fig. 2) are sym- 
metric with respect to the mirror plane that brings one 
octahedron into the other, while the d^^-y-^ are antisym- 
metric. Now any direct overlap between d wave functions 
in transition metal oxides is known to be very small. 
However one should not forget that the orbitals are in 
fact Wannier functions centered on the transition metal 
ions which extend in general to infinity to insure orthogo- 
nality, and which have a significant weight on neighbour- 
ing O 2p orbitals. In the case of the Wannier orbitals 
with symmetry dx2_y2^ this does not lead to any signif- 
icant transfer because the O 2p orbitals coupled to one 
of them are orthogonal to the dx2_y2 of the neighbour- 
ing octahedron. This is not strictly true here since the 
Ni-O-Ni angle is not exactly 90°, and also because the 
crystal field is not symmetric at the oxygen site, but still 
one expects the effective hopping to be very small. By 
contrast, the d^z'^_j.2 Wannier orbitals have weight on the 
O 2p orbitals above and below, and these O 2p orbitals 
have a standard tt overlap regardless of the actual local 
distortions of the octahedra. So this should give rise to 
a significant overlap between the Wannier functions with 
d^22_^2 symmetry. 

Beyond the actual value of the parameters, it is im- 
portant to emphasize that we have not adopted the same 
point of view as Mostovoy and Khomskiii, who have ne- 
glected any overlap between Ni orbitals, although it is 
allowed by symmetry. Further, they have assumed that 
Ni-O-Ni bonds make and angle of 90°, although the ac- 
tual angle is around 94 ° in LiNi02 and 96.4 ° in NaNi02, 
and they have neglected the role of crystal field at the 
oxygen site, known to produce antiferromagnetic cou- 
plings as shown by Dare et al2i. While the ferromag- 
netic coupling that comes out of these approximations is 
certainly relevant, the simplified Hamiltonian studied by 
Mostovoy and Khomskii leads to a purely ferromagnetic 
coupling, while the more general Hamiltonian studied in 
the present work exhibits a rich variety of phases which, 
we believe, might actually lead to the ultimate explana- 
tion of LiNi02 and NaNi02. 



A. LiNiOa 

In the case of LiNi02 , which undergoes neither a Jahn- 
Tcller distortion nor a magnetic phase transition upon 
lowering the temperature, we have to choose between 
two different realizations of RVB: the SU(4) phase A (for 
t ~ t'), and the fluctuating dimer phases C and C. Since 
we have argued that \t' /t\ ^ 1, we opt for the dimer 
phases. Actually, one should give preference to Phase 
C since t' /t is negative, but as we discussed these phases 
should better be considered as defining a domain in which 
the physics of the quantum dimer model (QDM) on the 
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triangular lattice might be relevant. The actual form of 
the effective QDM is not known yet, but it presumably 
will not be too far from the minimal model studied by 
Moessner and Sondhi2& since phase C is a staggered state 
and belongs to the ground state manifold of their model 
for large engouh repulsion between face-to-face dimers, 
while phase C is a maximally flippablc state and belongs 
to the ground state manifold in the limit of infinite at- 
traction between face-to-face dimers. Note however that 
Phase C is not the columnar state realized for finite at- 
traction in the minimal model, so differences are to be ex- 
pected. Still, close to the boundaries between the phases, 
one may speculate that an RVB phase will be present. 
Such a phase does not break any symmetry and could ex- 
plain the absence of any kind of ordering in LiNi02. Let 
us also note that the EXAFS results by Rougier et aim 
are also consistent with this proposal since the orbitals 
entering all these states are Jahn- Teller orbitals with two 
long bonds and four short bonds. If the system undergoes 
resonances between different states, this would produce a 
dynamic Jahn- Teller effect between these states, a situa- 
tion still leading to two long bonds and four short bonds 
on average. Due to some disorder, and/or to coupling to 
the lattice, the system might actually prefer to freeze in 
a non-periodic dimer covering of the triangular lattice, 
as suggested by Reynaud et aliS. Such a frozen, non- 
periodic state would also be consistent with the results 
of Ref.[||. 



B. NaNiOa 

As far as NaNi02 is concerned, the only potential can- 
didate is phase D since this is the only fcrro-orbital phase 
with Jahn- Teller orbitals consistent with the distortion 
that occurs at 480 K in that system. This phase has the 
largest boundary with phase C, a good point in view of 
the very similar structures of LiNi02 and NaNi02. As 
stated earlier, the effective model consists of weakly cou- 
pled AF chains, and the resulting magnetic structure will 
depend on the residual couplings. A thorough analysis 
of this point will require to go beyond the present cal- 
culation and is left for future investigation. But in any 
case, with some interlayer coupling, this is expected to 
lead to some kind of AF ordering at finite temperature, 
in agreement with experiments. Let us emphasize that, 
while simultaneous ferromagnetism and Jahn- Teller ac- 
tive fcrro-orbital order have been argued to be possible 
by Mostovoy and Khomskiii in the context of their sim- 
plified model, this seems to be impossible in the context 
of our microscopic model. Now, as far as experiments 
are concerned, the actual order is not known yet. It has 
been often assumed so far that this AF state consists of 
ferromagnetic planes coupled antiferromagnetically, but 
new, preliminary results seem to indicate that this can- 
not be the case^i, which opens the way for another type 
of antiferromagnet. 



C. Curie- Weiss constant 

At this stage, the essential problem when comparing 
our predictions to the experimental data for LiNi02 and 
NaNiOj^ is the sign of the Curie- Weiss constant, which is 
a measure of the average coupling. In both cases, it is fer- 
romagnetic if determined at not too high temperaturei^, 
while in our calculation, it is antiferromagnetic. This 
is not a very serious problem however. In deriving our 
model, we have only kept second order terms in the hop- 
ping t and t' between Wannier orbitals centered at Ni 
sites. This derivation, exactly similar to the derivation 
of kinetic exchange, neglects the ferromagnetic coupling 
that is always present due to the overlap of these Wan- 
nier functions. The precise form of this ferromagnetic 
coupling in the context of extended Wannier functions 
centered at the nickel site, which is the point of view 
adopted in the present paper, has not been derived yet 
but it is expected to be similar to the form derived by 
Mostovoy and Khomskii. In any case, we have checked 
that if wc include an ad hoc ferromagnetic term (the B' 
term in Eq. ((T7|) ) 

H = -JF^S,.Sj (40) 

to the Hamiltonian {Jp > 0), the phases discussed above 
remain stable in a region where the Curie- Weiss constant 
is ferromagnetic. More precisely, we have solved the self- 
consistent equations including such a term, which leads 
to the effective spin Hamiltonian: 

i/ = ^{(S..S,)/4+fc,^} (41) 

where h'^j = {hfj) — Jp, and the Curie- Weiss constant is 
given by: 

Ocw = ^^Y.h'., = lY.'^'. (42) 

<ij> 

where means summation over all first neighbours 

of a a given site while X]<y> means summing over 
three pairs of nearest-neighbours in three inequivalent 
directions. The results are summarized in figure El As 
announced, the Curie- Weiss temperature changes sign in- 
side phases C and D before one enters the ferromagnetic 
phase, and this occurs for values of Jp which are small 
enough to be physically relevant. 

VIII. CONCLUSION 

We have shown that a spin-orbital model on the tri- 
angular lattice with realistic parameters leads to a very 
rich physics. The presence of various important phases 
(SU(4), dimers and ferromagnetic) is confirmed for every 
cluster (4, 12 and 16 sites). Moreover it seems that this 
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given specific meaning to the claim that an RVB state 
seems to be at the origin of the magnetic properties of 
LiNi02. The underlying orbital structure corresponding 
to this RVB state is in agreement with the experimental 
observations. For the case of NaNi02, a new possible 
magnetic state has been investigated with an underlying 
orbital structure which still leads to a cooperative Jahn- 
Tcllcr distortion. A precise description of the low energy 
physics of the present model for the phases relevant for 
LiNi02 and NaNi02 requires other methods than those 
used in the present paper, but we are confident that the 
present analysis will set the stage for further investiga- 
tions. 



FIG. 17: 9cw as a function of Jf for C and D phases (see 
fig. 0: J/U = 0.064 and t' /t = -0.1 for the solid line (D 
phase), t' /t — 0.1 for the dashed line (C phase) 



model is able to provide a good description of the be- 
haviour of LiNi02 and NaNi02; and to explain the puz- 
zling difference between these two compounds. We have 
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